Systems And Methods For Tracking Objects

ABSTRACT

Systems and methods to track an object within an operating room with a camera unit. The camera unit includes a first optical sensor with sensing elements to sense light in a near-infrared spectrum and a second optical sensor to sense light in a visible light spectrum. A controller is in communication with the first and second optical sensors. The controller obtains, from the second optical sensor, data related to the object within the operating room. The controller modifies control of the sensing elements of the first optical sensor based on the data of the object obtained by the second optical sensor.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.17/972,625, filed Oct. 25, 2022, which is a continuation of U.S. patentapplication Ser. No. 17/317,191, filed on May 11, 2021, now U.S. Pat.No. 11,510,740, which is a continuation of U.S. patent application Ser.No. 16/441,645, filed Jun. 14, 2019, now U.S. Pat. No. 11,007,018, whichclaims priority to and all advantages of U.S. Provisional Patent App.No. 62/685,470, filed on Jun. 15, 2018, the entire contents of each ofthe aforementioned applications being hereby incorporated by reference.

TECHNICAL FIELD

The present disclosure relates generally to systems and methods fortracking objects.

BACKGROUND

Navigation systems assist users in locating objects. For instance,navigation systems are used in industrial, aerospace, and medicalapplications. In the medical field, navigation systems assist surgeonsin precisely placing surgical instruments relative to a patient'sanatomy. Surgeries in which navigation systems are used includeneurosurgery and orthopedic surgery. Typically, the instrument and theanatomy are tracked together with their relative movement shown on adisplay.

Navigation systems may employ light signals, sound waves, magneticfields, RF signals, etc., in order to track the position and/ororientation of objects. Often the navigation system cooperates withtracking devices attached to the object being tracked. The navigationsystem includes a localizer to determine a position of the trackingdevices, and ultimately to determine a position and/or orientation ofthe object. The navigation system monitors movement of the object viathe tracking devices.

Frequently, localizers determine the position of tracked objects bysampling reflections or emissions of light from trackers attached to thetracked objects at a defined sampling rate. For example, some localizerssample light from the trackers at about 60 Hertz (Hz). Other localizersmay sample light from the trackers up to about 335 Hz. Two-dimensionalsensors employed by a localizer require processing a large volume ofdata. The localizer includes optical sensors, each sensor having anarray of sensing elements and each sensing element having a range ofvalues corresponding to the incident energy on the element. Processingan image from these values requires reading out the information fromeach element in sequence, that is—the readout processing cannot beparallelized. Sensors suitable for use in a localizer have a high numberof elements with large ranges of values, so processing the sensorreadout becomes a bottleneck and is a limiting factor in improving thesampling rate for tracking technologies. These sampling rates may beinsufficient to detect rapid movement of the tracked objects adequately.Similarly, a low sampling rate may be insufficient to detect smallmovements of the tracked objects.

Increasing the sampling rate to address these shortcomings introducesits own challenges. For example, an increase in the sampling rate cansubstantially increase a processing workload for a processor that istasked with analyzing the sampled signals to determine the presence andpose of the tracked objects. In some situations, the processor may notbe able to keep up with the rate and number of sampled signals receivedfrom the trackers and may thus fail to detect changes in pose of thetracked objects. As described above, the readout from the sensor itselfis a limiting factor in improving processing time as more data from theprocessor increases the workload of the processor.

The present disclosure addresses one or more of the above-describedproblems.

SUMMARY

According to a first aspect, a surgical navigation system is providedfor tracking an object within an operating room, the surgical navigationsystem comprising: a camera unit comprising: a housing; a first opticalsensor coupled to housing and comprising sensing elements adapted tosense light in a near-infrared spectrum; a second optical sensor coupledto the housing and being adapted to sense light in a visible lightspectrum; and a controller in communication with the first and secondoptical sensors, and wherein the controller is configured to: obtain,from the second optical sensor, data related to the object within theoperating room; and modify control of the sensing elements of the firstoptical sensor based on the data of the object obtained by the secondoptical sensor.

According to a second aspect, a method is provided of operating asurgical navigation system for tracking an object within an operatingroom, the surgical navigation system comprising a camera unit comprisinga housing, a first optical sensor coupled to housing and comprisingsensing elements adapted to sense light in a near-infrared spectrum, asecond optical sensor coupled to the housing and being adapted to senselight in a visible light spectrum, and a controller in communicationwith the first and second optical sensors, the method comprising thecontroller performing the steps of: obtaining, from the second opticalsensor, data related to the object within the operating room; andmodifying control of the sensing elements of the first optical sensorbased on the data of the object obtained by the second optical sensor.

According to a third aspect, a camera unit is provided for tracking anobject within an operating room, the camera unit comprising: a firstoptical sensor comprising sensing elements adapted to sense light in anear-infrared spectrum; a second optical sensor adapted to sense lightin a visible light spectrum; and a controller in communication with thefirst and second optical sensors, and wherein the controller isconfigured to: obtain, from the second optical sensor, data related tothe object within the operating room; and modify control of the sensingelements of the first optical sensor based on the data of the objectobtained by the second optical sensor.

BRIEF DESCRIPTION OF THE DRAWINGS

Advantages of the present disclosure will be readily appreciated as thesame becomes better understood by reference to the following detaileddescription when considered in connection with the accompanying drawingswherein:

FIG. 1 is a perspective view of a navigation system being used inconjunction with a robotic system.

FIG. 2 is a schematic view of the navigation system.

FIG. 3 is schematic view of the coordinate systems used in thenavigation system.

FIG. 4 is a schematic view showing the relationship between opticalsensors and a working space.

FIG. 5 is a schematic view of an arrangement of sensing elements.

FIG. 6 is a representative view of light incident on an optical sensor.

FIG. 7 is a schematic representation of observed volume by a cameraunit.

FIG. 8 is a perspective view of a navigation system in an alternativeembodiment.

FIG. 9 is a block diagram of a camera unit that may be used with thenavigation system.

FIG. 10 is a perspective view of an optical filter that may be used tofilter light received from markers within the navigation system.

FIG. 11 is a block diagram of a bit mask array that may be used toselect a subset of sensor elements used by the navigation system.

FIG. 12 is a block diagram of an array of gating devices that may beused to select a subset of sensing elements used by the navigationsystem.

FIG. 13 is a flow diagram of a method of tracking objects within anoperating room.

DETAILED DESCRIPTION

Referring to FIG. 1 , a surgical system 10 is illustrated for performingsurgery on a patient. The version shown in FIG. 1 includes a surgicalnavigation system 20. The surgical navigation system 20 is shown in asurgical setting such as an operating room of a medical facility. Thesurgical navigation system 20 is set up to track movement of variousobjects in the operating room. Such objects include, for example, asurgical instrument 22, a femur F of the patient, a tibia T of thepatient, and/or a robotic manipulator 56. The surgical navigation system20 tracks these objects for purposes of displaying their relativepositions and orientations to the surgeon and, in some cases, forpurposes of controlling or constraining movement of the surgicalinstrument 22 relative to virtual cutting boundaries associated with thefemur F and tibia T.

The surgical navigation system 20 includes a computer cart assembly 24that houses a navigation computer 26. A navigation interface is inoperative communication with the navigation computer 26. The navigationinterface includes a first display 28 adapted to be situated outside ofthe sterile field and a second display 29 adapted to be situated insidethe sterile field. The displays 28, 29 are adjustably mounted to thecomputer cart assembly 24. First and second input devices (not shown)such as a keyboard and mouse can be used to input information into thenavigation computer 26 or otherwise select/control certain aspects ofthe navigation computer 26. Other input devices are contemplatedincluding a touch screen 30, gesture control, or voice-activation.

A localizer 34 communicates with the navigation computer 26. In theembodiment shown, the localizer 34 is an optical localizer and includesa camera unit 36 (one example of a sensing device). The camera unit 36has an outer casing 38 that houses one or more optical position sensors40. The optical sensors 40 may be rigidly mounted to a common supportstructure. The outer casing 38 may provide the common support structurefor the optical sensors 40. Alternatively, a rigid support structurecommon to the optical sensors 40 may be encased by, but distinct from,the outer casing 38. As illustrated in FIG. 1 , the optical sensors 40are disposed at opposite ends of the elongated camera unit 36, such thatthe optical sensors are arranged stereoscopically and separated by aseparation distance. Representative separation distances may be greaterthan about 6 inches, greater than about 8 inches, greater than about 12inches, or greater than about 24 inches. Larger separation distances mayimprove the three-dimensional depth perception of the system at the costof larger component size. The larger the size of the camera unit 36 mayincrease the difficulty of arranging the camera unit 36 to maintain anobstruction-free view of the target space. In some embodiments at leasttwo optical sensors 40 are employed. The optical sensors 40 are capableof variable attenuation of radiant energy, for example, light, intosignals as small bursts of electrical current that convey information.

The camera unit 36 may also include a video camera 41 or otheradditional sensing device. The video camera 41 may be one or morefull-color optical sensors, including one or more charge-coupled devices(CCD), complimentary metal-oxide semiconductor (CMOS) active-pixelsensors, and the like. The video camera 41 may provide real-time or lowlatency video monitoring of the surgical operation. The video camera 41may include similar or different optical sensing technology as thoseemployed in the optical sensors 40. For example, the optical sensors 40may be adapted to sense light in the infrared or near-infrared spectrum,while the video camera 41 may be adapted to sense light in the visiblespectrum. In an alternative, the optical sensors 40 and the video camera41 may include similar CMOS sensors adapted to sense light in thevisible spectrum.

In some embodiments at least two optical sensors 40 are employed,alternatively, three or four optical sensors 40 may be employed. Theoptical sensors 40 may be separate CCDs. In some embodiments,two-dimensional CCDs are employed and in other embodiments,one-dimensional CCDs are employed. In some cases, the two,two-dimensional optical sensors 40 are arranged for stereoscopicoperation. In some embodiments, a single optical sensor 40 may beprovided in combination with depth sensors, laser range finders, and thelike. In some other embodiments, a single optical sensor 40 may beemployed if a sufficient number of fiducials are within the sensor view,for example, at least four fiducials, and the geometry of the fiducialdistribution is known. It should be appreciated that in otherembodiments, separate camera units, each with a separate CCD, or two ormore CCDs, could also be arranged around the operating room. The opticalsensors 40 may include CCDs capable of detecting infrared (IR) radiantenergy. In alternative embodiments, the optical sensors 40 may employother technology, including, but not limited to, complimentarymetal-oxide semiconductor (CMOS) active-pixel sensors, and the like.

The camera unit 36 may be mounted on an adjustable arm or otherarticulated support structure of the cart assembly 24 to selectivelyposition the localizer 34 with a, preferably unobstructed, field of viewof the target space including the surgical setting within which will bethe patient anatomy and trackers, as discussed below. In someembodiments, the camera unit 36 is adjustable in at least one degree offreedom by rotating about a rotational joint. In other embodiments, thecamera unit 36 is adjustable about two or more degrees of freedom.

The camera unit 36 includes a camera controller 42 in communication withthe optical sensors 40 to receive signals from the optical sensors 40.The camera controller 42 may be in further communication with the videocamera 41. Alternatively, a separate controller from the cameracontroller 42 may be provided as a machine vision controller tocommunicate video information from the video camera 41 to the navigationcomputer 26. In one embodiment, the machine vision controller incommunication with the video camera 41 and the navigation controller areintegrally provided on a single printed-circuit board assembly, such asis illustrated in FIG. 1 . The integrated controller handling navigationand machine vision will be referred to as the camera controller 42.

The camera controller 42 communicates with the navigation computer 26through either a wired or a wireless connection (not shown). One suchconnection may be an IEEE 1394 interface, which is a serial businterface standard for high-speed communications and isochronousreal-time data transfer. The connection could also use a companyspecific protocol. In other embodiments, the optical sensors 40 maycommunicate directly with the navigation computer 26, such that thenavigation computer incorporates the functionality of, and thus operatesas, the camera controller 42. Processing of the signals from the opticalsensors 40 and the video camera 41 may occur at the camera controller42. Alternatively, the camera controller 42 may communicate the signalsto the navigation computer 26 for processing for both navigation andmachine vision.

The navigation computer 26 can be a personal computer or laptopcomputer. Navigation computer 26 has the display 28, central processingunit (CPU) and/or other processors, memory (not shown), and storage (notshown). The navigation computer 26 is loaded with software as describedbelow. The software converts the signals received from the camera unit36 into data representative of the position and orientation of theobjects being tracked. Additionally, the software converts the signalsreceived from the camera unit 36 into data that can identify theobjects, such as through object recognition from the video camera 41.Position and orientation signals and/or data are transmitted to thenavigation computer 26 for purposes of tracking objects. In analternative, all of the computer processing components and functionalitymay be integrated into a single processing units, or may be distributedbetween or among multiple processing units. Moreover, although describedas taking place at a particular computer or controller in the presentdisclosure, it will be appreciated by one of skill in the art that anyprocessing tasks may take place or be performed by other computers orcontrollers. The computer cart assembly 24, display 28, and camera unit36 may be like those described in U.S. Pat. No. 7,725,162 toMalackowski, et al. issued on May 25, 2010, entitled “Surgery System,”hereby incorporated by reference.

The surgical system 10 illustrated in FIG. 1 includes a plurality oftracking devices 44, 46, 48, also referred to herein as trackers. In theillustrated embodiment, one tracker 44 is coupled to the femur F of thepatient and another tracker 46 is coupled to the tibia T of the patient.Trackers 44, 46 may be attached to the femur F and tibia T in the mannershown in U.S. Pat. No. 7,725,162 to Malackowski, et al. issued on May25, 2010, entitled “Surgery System,” hereby incorporated by reference.Trackers 44, 46 could also be mounted like those shown in U.S. PatentApplication Publication No. 2014/0200621, published on Jul. 17, 2014,entitled, “Navigation Systems and Methods for Indicating and ReducingLine-of-Sight Errors,” hereby incorporated by reference herein. Inadditional embodiments, a tracker (not shown) is attached to the patellato track a position and orientation of the patella. In otherembodiments, the trackers 44, 46 could be mounted to other tissue typesor parts of the anatomy according to the needs of a particularoperation.

An instrument tracker 48 is coupled to the surgical instrument 22. Theinstrument tracker 48 may be integrated into the surgical instrument 22during manufacture or may be separately mounted to the surgicalinstrument 22 in preparation for the surgical procedures. The workingend of the surgical instrument 22, which is being tracked by virtue ofthe instrument tracker 48, may be an energy applicator EA such as arotating bur, saw blade, electrical ablation device, or the like. Theenergy applicator EA may be a separate component such as a bur, sawblade, ablator, or the like that is releasably connected to a handpieceof the surgical tool 22 or may be integrally formed with the handpiece.

The trackers 44, 46, 48 may be active trackers or passive trackers.Active trackers require a power source and have an array of fiducials(also referred to as tracking elements or markers) that activelygenerate and emit radiation in a wavelength detectable by the opticalsensors 40. The fiducials of an active tracker may be a light emittingdiode (LED), including, for example, an infrared LED. The array ofactive fiducials may be “always on” or may be operative to selectivelyfire, that is emit radiation, according to and in response to commandsfrom the surgical navigation system 20. In such selective-fire activetrackers, the tracker may communicate by way of a wired or a wirelessconnection with the navigation computer 26 of surgical navigation system20. In alternative embodiments, the tracker may include passivetrackers. That is, the array of passive trackers focus or reflectambient radiation or radiation that has been emitted into the targetspace, for example by one or more infrared LEDs provided on the cameraunit 36 or elsewhere associated with the surgical system 10. The activetracker may be battery powered with an internal battery or may haveleads to receive power through the navigation computer 26, which, likethe camera unit 36, may receive external power. The passive trackerarray typically does not require a power source.

In the embodiment shown, the surgical instrument 22 is attached to asurgical manipulator 56. Such an arrangement is shown in U.S. Pat. No.9,119,655, issued Sep. 1, 2015, entitled, “Surgical Manipulator Capableof Controlling a Surgical Instrument in Multiple Modes”, the disclosureof which is hereby incorporated by reference.

In other embodiments, the surgical instrument 22 may be manuallypositioned by only the hand of the user, without the aid of any cuttingguide, jig, or other constraining mechanism such as a manipulator orrobot. Such a surgical instrument is described in U.S. Pat. No.9,707,043, issued Jul. 18, 2017, the disclosure of which is herebyincorporated by reference.

The optical sensors 40 of the localizer 34 receive signals from thetrackers 44, 46, 48. In the illustrated embodiment, the trackers 44, 46,48 are active trackers. In this embodiment, each tracker 44, 46, 48 hasat least three active tracking elements or markers for transmittinglight signals to the optical sensors 40. The active markers can be, forexample, light emitting diodes or LEDs 50 transmitting light, such asinfrared light. The optical sensors 40 preferably have sampling rates of100 Hz or more, more preferably 300 Hz or more, and most preferably 500Hz or more. In some embodiments, the optical sensors 40 have samplingrates of 8000 Hz. The sampling rate is the rate at which the opticalsensors 40 receive light signals from sequentially fired LEDs 50. Insome embodiments, the light signals from the LEDs 50 are fired atdifferent rates for each tracker 44, 46, and 48. In other embodiments,the LEDs 50 are always-on, active tracking elements or markers.

Initially, the objects to be located are viewed by the optical sensors40 and video camera 41 and identified. The objects may be identified byselecting the objects to be tracked using an input device connected tothe navigation computer 26. The navigation computer 26 may storedetailed information regarding numerous objects in memory or datastorage on the navigation computer 26 and the user may be able tomanually select the objects to be tracked from a database of objects.

Additionally, or alternatively, the navigation computer 26 may identifythe objects to be tracked based on a pre-operative surgical plan. Inthis case, the navigation computer 26 may have a preset list of workflowobjects that may be used in the pre-scripted surgical workflow. Thenavigation computer 26 may actively search for and locate the workflowobjects using software in the image data provided by the optical sensors40 or video camera 41. For instance, groups of pixels associated withdifferent sizes and shapes of the various objects may be stored in thenavigation computer 26. By selecting/identifying the objects to belocated/tracked, the software identifies the corresponding group ofpixels and the software then operates to detect like groups of pixelsusing conventional pattern recognition technology.

Additionally, or alternatively, the objects to be located/tracked can beidentified using an interface in which one of the participants outlinesor selects the objects to be tracked on one or more of the displays 28,29. For instance, images taken by the optical sensors 40, or videocamera 41, of the surgical site may be displayed on one or more of thedisplays 28, 29 (and/or other displays). The participant then, using amouse, digital pen, or the like, traces objects to be located/tracked onthe display 28 and/or 29. The software stores the pixels associated withthe object that was traced into its memory. The participant (or otheruser) may identify each object by a unique identifier such as naming theobject using the software so that the saved group of pixels may beassociated with the unique identifier. Multiple objects could be storedin this manner. The navigation computer 26 utilizes conventional patternrecognition and associated software to later detect these objects. Thenavigation system 20 is able to detect movement of these objects bycontinuously taking images, reviewing the images, and detecting movementof the groups of pixels associated with the objects.

The objects to be tracked may be initially located and registered usinga navigation pointer P. For example, the navigation pointer P may havean integrated tracker PT. The navigation computer 26 may store initialdata corresponding to a location of the tip of the pointer P relative tothe tracker PT such that the navigation system 20 is able to locate andtrack the tip of the pointer P in the localizer coordinate system LCLZ.Accordingly, prior to the start of the surgical procedure, once all theobjects are located in their desired locations, one of the participantsmay touch all of the objects with the pointer P, while identifying theobjects in the navigation system 20 using one of the input devicesdescribed above. For example, when the participant touches the surgicalinstrument 22 with the tip of the pointer P, the participant maysimultaneously trigger collection of that point in the localizercoordinate system LCLZ (via another input device, such as a foot pedal).When the point is collected, the participant can also enter into thenavigation software the identity of the object (via typing, pull-downselection from a list of objects, etc.).

The machine vision system is incorporated into the navigation system 20.More specifically, the machine vision system may include a machinevision controller that is coupled to the navigation computer 26, or maybe integrated with the camera controller 42. The machine vision systemincludes one or more machine vision cameras coupled to the machinevision controller, such as the video camera 41 coupled to the cameracontroller 42. While one video camera 41 is illustrated in FIG. 1 , itshould be recognized that any suitable number of video cameras 41 orother optical sensors may be included within the machine vision system.The video cameras 41 may be CCDs or CMOS sensor-based cameras or otherforms of machine vision camera. In some cases, the video cameras arearranged for stereoscopic operation, or single cameras combined withdepth sensors, laser range finders, and the like, may be used.

Machine vision can identify and locate various objects in the operatingroom. The video camera 41 (and in some cases, depth sensors) can bearranged to determine 3-D positions and/or orientations of the objectsin a machine vision coordinate system. In the example illustrated inFIG. 1 , the video camera 41 providing machine vision is rigidlysupported with the optical sensors 40 in a known and predefinedrelationship according to the manufacturing of the camera unit 34. Thevideo camera 41 and optical sensors 40 are arranged so that theirfield-of-view encompasses the objects in the operating room. By way ofnon-limiting example, the objects may include a robotic manipulator 56,one or more surgical instruments 22, portions of the patient's anatomy(e.g., tibia T and femur F), and/or any other suitable object. Inproviding an integrated camera unit 34, the need to operate in andtransform between a machine vision coordinate system and a navigationcoordinate system can be alleviated or eliminated. More specifically,the machine vision coordinate system and the navigation coordinatesystem may be the same coordinate system. In the embodiment described inFIG. 1 , the machine vision coordinate system and the navigationcoordinate system are the same and collectively identified as thelocalizer coordinate system LCLZ, described in more detail below.

Initially, the objects to be located are viewed by the video camera 41and optical sensors 40 and identified. The objects may be identified byselecting the objects to be tracked using an input device connected tothe navigation computer 26. The navigation computer 26 may storedetailed information regarding numerous objects in memory on navigationcomputer 26 or the camera controller 42 and the user may be able tomanually select the objects to be tracked from a database of objects.

Additionally, or alternatively, the machine vision controller 14 mayidentify the objects to be tracked based on a pre-operative surgicalplan. In this case, the navigation computer 26 may have a preset list ofworkflow objects that may be used in the pre-scripted surgical workflow.The navigation computer 26 may actively search for and locate theworkflow objects using machine vision software. For instance, groups ofpixels associated with different sizes and shapes of the various objectsmay be stored in the navigation computer 26. By selecting/identifyingthe objects to be located/tracked, the machine vision softwareidentifies the corresponding group of pixels and the machine visionsoftware then operates to detect like groups of pixels usingconventional pattern recognition technology.

Additionally, or alternatively, the objects to be located/tracked can beidentified using an interface in which one of the participants outlinesor selects the objects to be tracked on one or more of the displays 28,29. For instance, images taken by the video camera 41 or optical sensors40 from overhead the surgical site may be displayed on one or more ofthe displays 28, 29 (and/or other displays). The participant then, usinga mouse, digital pen, or the like, traces objects to be located/trackedon the display 28 and/or 29. The machine vision software stores thepixels associated with the object that was traced into its memory. Theparticipant (or other user) may identify each object by a uniqueidentifier such as naming the object using the machine vision softwareso that the saved group of pixels may be associated with the uniqueidentifier. Multiple objects could be stored in this manner. Thenavigation computer 26 utilizes conventional pattern recognition andassociated software to later detect these objects in the image dataprovided by the video camera 41 or the optical sensors 40.

The navigation system 20 is able to detect movement of these objects bycontinuously taking images, reviewing the images, and detecting movementof the groups of pixels associated with the objects. In some cases,location information from the camera controller 42 for the objects canbe transmitted to the navigation computer 26. Likewise, locationinformation from the navigation computer 26 can be transmitted from thenavigation computer 26 to the camera controller 42.

After the navigation system 20 identifies and locates any desiredobjects within the operating room, the navigation computer 26 maytransmit the location and identity of the objects to the cameracontroller 42. The navigation computer 26 and/or the camera controller42 uses the location and identity of the objects to selectively adjustthe localizer 34, including the data output from one or more of theoptical sensors 40 or video camera 41 to focus on the portion of theoperating room that includes the objects. Thus, the navigation system 20may disregard other areas of the operating room as described more fullyherein, thus improving a processing and tracking efficiency of thenavigation system 20.

Referring to FIG. 2 , a schematic view of a control system forcontrolling the surgical navigation system 20 and robotic surgicaldevice 56 is shown. In this schematic, each of the LEDs 50 areillustrated connected to a tracker controller 62 located in a housing(not shown) of the associated tracker 44, 46, 48 that transmits/receivesdata to/from the navigation computer 26 and/or camera controller 42. Inone embodiment, the tracker controllers 62 transmit data through wiredconnections with the navigation computer 26. In other embodiments, awireless connection may be used. In these embodiments, the navigationcomputer 26 has a transceiver (not shown) to receive the data from thetracker controller 62. In other embodiments, the trackers 44, 46, 48 mayhave passive markers (not shown), such as reflectors that reflect light,for example, light emitted by an LED provided on camera unit 36. Forexample, the camera unit 36 may include complementary emitters in awavelength to which the optical sensors 40 are sensitive. The reflectedlight is then received by the optical sensors 40. In some embodiments,the trackers 44, 46, 48 may also include a gyroscope sensor 60 andaccelerometer 70, such as the trackers shown in U.S. Pat. No. 9,008,757to Wu, et al., issued on Apr. 14, 2015, entitled, “Navigation SystemIncluding Optical and Non-Optical Sensors,” the entire disclosure ofwhich is hereby incorporated by reference. These additional sensors 60,70, may provide information to the navigation computer 26 for use by thenavigation computer 26 to determine or track the trackers' 44, 46, 48position or orientation.

The navigation computer 26 includes a navigation processor 52. It shouldbe understood that the navigation processor 52 could include one or moreprocessors to control operation of the navigation computer 26, mayperform one or more navigation functions, and may perform one or moremachine vision functions. The processors can be any type ofmicroprocessor or multi-processor system. The term “processor” is notintended to limit the scope of the invention to a single processor or toany particular function.

As illustrated in FIG. 2 , the camera unit 36 receives optical signals53 from the LEDs 50 of the trackers 44, 46, 48 and outputs to theprocessor 52 signals relating to the position of the LEDs 50 of thetrackers 44, 46, 48 relative to the localizer 34. Based on the receivedoptical (and non-optical signals in some embodiments), navigationprocessor 52 generates data indicating the relative positions andorientations of the trackers 44, 46, 48 relative to the localizer 34.

Prior to the start of the surgical procedure, additional data are loadedinto the navigation processor 52. Based on the position and orientationof the trackers 44, 46, 48 and the previously loaded data, navigationprocessor 52 determines the position of the working end of the surgicalinstrument 22 (e.g., the centroid of a surgical bur) and the orientationof the surgical instrument 22 relative to the tissue against which theworking end is to be applied. In some embodiments, navigation processor52 forwards the data to a manipulator controller 54. The manipulatorcontroller 54 can then use the data to control a robotic manipulator 56as described in U.S. Pat. No. 9,119,655 to Bowling, et al., incorporatedabove.

The navigation processor 52 also generates image signals that indicatethe relative position of the surgical instrument working end to thetissue. These image signals are applied to the displays 28, 29. Displays28, 29, based on these signals, generate images that allow the surgeonand staff to view the relative position of the surgical instrumentworking end to the surgical site. The displays, 28, 29, as discussedabove, may include a touch screen 30 or other input/output device thatallows entry of commands.

In the embodiment shown in FIG. 1 , the surgical tool 22 forms part ofan end effector of the manipulator 56. The manipulator 56 has a base 57,a plurality of links 58 extending from the base 57, and a plurality ofactive joints (not numbered) for moving the surgical tool 22 withrespect to the base 57. The links 58 may form a serial arm structure asshown in FIG. 1 , a parallel arm structure (not shown), or othersuitable structure. The manipulator 56 has the ability to operate in amanual mode in which a user grasps the end effector of the manipulator56 in order to cause movement of the surgical tool 22 (e.g., directly,through force/torque sensor measurements that cause active driving ofthe manipulator 56, or otherwise) or a semi-autonomous mode in which thesurgical tool 22 is moved by the manipulator 56 along a predefined toolpath (e.g., the active joints of the manipulator 56 are operated to movethe surgical tool 22 without requiring force/torque on the end effectorfrom the user). An example of operation in a semi-autonomous mode isdescribed in U.S. Pat. No. 9,119,655 to Bowling, et al., incorporatedabove. A separate tracker (not shown) may be attached to the base 57 ofthe manipulator 56 to track movement of the base 57.

The manipulator controller 54 may have a central processing unit (CPU)and/or other manipulator processors, memory (not shown), and storage(not shown). The manipulator controller 54, also referred to as amanipulator computer, is loaded with software. The manipulatorprocessors could include one or more processors to control operation ofthe manipulator 56. The manipulator 56 may be in the form of aconventional robotic system or other conventional machining apparatus,and thus the components thereof shall not be described in detail. In oneembodiment, when the manipulator 56 is operated in the semi-autonomousmode, the manipulator 56 is capable of moving the surgical tool 22 freeof operator assistance. Free of operator assistance may mean that anoperator/user does not physically contact the surgical tool 22 to movethe surgical tool 22. Instead, the operator may use some form of remotecontrol to control starting and stopping of movement. For example, theoperator may hold down a button of the remote control to start movementof the surgical tool 22 and release the button to stop movement of thesurgical tool 22.

In the manual mode, in one embodiment, the operator physically contactsthe end effector to cause movement of the surgical tool 22. Themanipulator controller 54 can use the position and orientation data ofthe surgical tool 22 and the patient's anatomy to control themanipulator 56 as described in U.S. Pat. No. 9,119,655 to Bowling, etal., incorporated above.

The manipulator controller 54 determines the desired location to whichthe surgical tool 22 should be moved. Based on this determination, andinformation relating to the current location (e.g., pose) of thesurgical tool 22, the manipulator controller 54 determines the extent towhich each of the plurality of links 58 needs to be moved in order toreposition the surgical tool 22 from the current location to the desiredlocation. The data regarding where the plurality of links 58 are to bepositioned is forwarded to joint motor controllers (not shown) (e.g.,one for controlling each motor) that control the active joints of themanipulator 56 to move the plurality of links 58 and thereby move thesurgical tool 22 from the current location to the desired location.

Referring to FIG. 3 , tracking of objects is generally conducted withreference to a localizer coordinate system LCLZ. The localizercoordinate system has an origin and an orientation (a set of x-, y-, andz-axes). During the procedure, one goal is to keep the localizercoordinate system LCLZ in a known position. An accelerometer (not shown)mounted to the camera unit 36 may be used to track sudden or unexpectedmovement of the localizer coordinate system LCLZ, as may occur when thecamera unit 36 is inadvertently bumped by surgical personnel.

Each tracker 44, 46, 48 and object being tracked also has its owncoordinate system separate from localizer coordinate system LCLZ.Components of the navigation system 20 that have their own coordinatesystems are the bone trackers 44 and 46, and the instrument tracker 48.These coordinate systems are represented as, respectively, bone trackercoordinate systems BTRK1 and BTRK2, and instrument tracker coordinatesystem TLTR.

Navigation system 20, through the localizer 34, monitors the positionsof the femur F and tibia T of the patient by monitoring the position ofbone trackers 44, 46 coupled to bone. The femur coordinate system isFBONE and the tibia coordinate system is TBONE, which are the coordinatesystems of the bones to which the bone trackers 44, 46 are coupled.

Prior to the start of the procedure, pre-operative images of the femur Fand tibia T are generated (or of other tissues in other embodiments).These images may be based on MRI scans, radiological scans or computedtomography (CT) scans of the patient's anatomy. These images are mappedto the femur coordinate system FBONE and tibia coordinate system TBONEusing well-known methods in the art. These images are fixed in the femurcoordinate system FBONE and tibia coordinate system TBONE. As analternative to taking pre-operative images, plans for treatment can bedeveloped in the operating room (OR) from kinematic studies, bonetracing, and other methods.

During an initial phase of the procedure, the bone trackers 44, 46 arecoupled to the bones of the patient. The pose (position and orientation)of coordinate systems FBONE and TBONE must be mapped to coordinatesystems BTRK1 and BTRK2, respectively. Given the fixed relationshipbetween the bones and their bone trackers 44, 46, positions andorientations of the femur F and tibia T in the femur coordinate systemFBONE and tibia coordinate system TBONE must be transformed to the bonetracker coordinate systems BTRK1 and BTRK2 so the camera unit 36 is ableto track the femur F and tibia T by tracking the bone trackers 44, 46.This pose-describing data are stored in memory integral with bothmanipulator controller 54 and navigation processor 52.

The working end of the surgical instrument 22 (also referred to asenergy applicator distal end) has its own coordinate system EAPP. Theorigin of the coordinate system EAPP may represent a centroid of asurgical cutting bur, for example. The pose of coordinate system EAPPmust be fixed to the pose of instrument tracker coordinate system TLTRbefore the procedure begins. Accordingly, the poses of these coordinatesystems EAPP, TLTR relative to each other must be determined in thenavigation computer 26. The pose-describing data are stored in memoryintegral with both manipulator controller 54 and navigation processor52.

Referring back to FIG. 2 , a localization engine 100 is a softwaremodule that may be included within the navigation system 20. Componentsof the localization engine 100 may execute on navigation processor 52.In some embodiments, however, the localization engine 100 may execute onthe manipulator controller 54 or camera controller 42.

Localization engine 100 receives as inputs the optically based signalsfrom the camera controller 42 and, in some embodiments, thenon-optically based signals from the tracker controller 62. Based onthese signals, localization engine 100 determines the pose of the bonetracker coordinate systems BTRK1 and BTRK2 in the localizer coordinatesystem LCLZ. Based on the same signals received for the instrumenttracker 48, the localization engine 100 determines the pose of theinstrument tracker coordinate system TLTR in the localizer coordinatesystem LCLZ.

The localization engine 100 forwards the signals representative of theposes of trackers 44, 46, 48 to a coordinate transformer 102. Coordinatetransformer 102 is a navigation system software module that runs onnavigation processor 52. Coordinate transformer 102 references the datathat defines the relationship between the pre-operative images of thepatient and the bone trackers 44, 46. Coordinate transformer 102 alsostores the data indicating the pose of the working end of the surgicalinstrument relative to the instrument tracker 48.

During the procedure, the coordinate transformer 102 receives the dataindicating the relative poses of the trackers 44, 46, 48 to thelocalizer 34. Based on these data and the previously loaded data, thecoordinate transformer 102 generates data indicating the relativeposition and orientation of the coordinate system EAPP, the machinevision coordinate system MV, and the bone coordinate systems, FBONE andTBONE to the localizer coordinate system LCLZ.

As a result, coordinate transformer 102 generates data indicating theposition and orientation of the working end of the surgical instrument22 relative to the tissue (e.g., bone) against which the instrumentworking end is applied. Image signals representative of these data areforwarded to displays 28, 29 enabling the surgeon and staff to view thisinformation. In certain embodiments, other signals representative ofthese data can be forwarded to the manipulator controller 54 to guidethe manipulator 56 and corresponding movement of the surgical instrument22.

In a similar manner, other trackers may be coupled to any other suitableobject to be tracked within the operating room, and each object andassociated tracker may be registered to the localizer coordinate systemLCLZ as described above.

Referring to FIG. 4 , a representation of the relationship of theoptical sensors 40 to the target space is represented. In theillustrated embodiment, two, two-dimensional optical sensors 40 arearranged for stereoscopic operation, mounted to a common supportstructure 72 and separated by a separation distance D. The commonsupport structure 72 may be enclosed within housing 38 (shown in FIG. 1). The optical sensors 40 are arranged with a view of the working space74. Within the working space 74, an element is provided as a region ofinterest 76. In a surgical operation, the region of interest 76 may be aparticular area of a patient's anatomy upon which the procedure isfocused. The region of interest 76 may encompass the entirety of theoptical sensors' 40 field of view, or alternatively, may be a portion ofthe full field of view 74.

Referring to FIG. 5 , a representation of the optical sensor's 40 fieldof view of the working space 74 is shown projected as it would beincident on the sensor elements of the optical sensor 40, with aparticular region of interest 76 also illustrated. In some embodiments,the optical sensors 40 include a printed circuit board assembly (PCBA)having an array of charge-coupled sensor elements. Each sensor elementmay be uniquely identifiable according to an addressable location on thesensor. For example, the sensor elements may be identifiable in anx-coordinate and y-coordinate according to the number of rows and numberof columns of elements on the sensor. Specifically, a first sensorelement 78 in an arbitrary top-left corner may be identified with thecoordinate (1, 1), while the last sensor element 80 in the opposite,bottom-right corner may be identified with the coordinate (1000, 1000),for a sensor having 1,000 rows and 1,000 columns of individual sensorelements. In this example, the sensor would therefore be characterizedas a 1 megapixel (MP) optical sensor, having one million active sensorelements. Each sensor element corresponds to one pixel of informationcontributing to the output of the sensor. The sensor array may typicallyform a rectangular or square array.

The region of interest 76 is present within a subset of the sensorelements. The region of interest 76 may be located by identifying abeginning pixel 82 and an ending pixel 84 on the sensor on which theregion of interest 76 acts. As shown in FIG. 5 , the region of interestmay be located across the pixels (201, 301) (shown at 82 in FIG. 5 ) to(800, 700) (shown at 84 in FIG. 5 ). This region forms an array having awidth 86 of 600 pixels wide and having a height 88 of 400 pixels tall.The region of interest therefore occupies 0.24 MP of the 1 MP sensor. Inreading out information from the sensor, defined by the range of sensorelements within the region of interest 76, the data processing load istherefore 24% of the data processing load of the full range of activesensor elements in the working space 74. A reduction in the dataprocessing load provides a corresponding increase in the data processingcycle speed. That is, each cycle of processing the image data consumesless time as the amount of data to process is reduced.

Supported within the housing 38 and between the sensor PCBA of eachoptical sensor 40 and the physical volume in which the surgeon operates,an optic element, such as a lens, is provided to focus incident radiantenergy onto the sensor elements. In some embodiments, a single lens isprovided macroscopically over the PCBA, and in other embodiments,microlenses may be provided over each individual sensor element. Thelens may be static, or may be adjustable to more precisely focus theenergy onto the sensor elements. This relationship is shown in FIG. 6 .The total array of sensor elements provided on the PCBA is representedby the rectangular area 90. In the example shown, a single lens focusesenergy onto a substantially circular area 92 of sensor elements. Thesensor elements disposed outside this circular area may be blocked bythe housing 38, and thus considered inactive. The camera controller 42may be configured to exclude any inactive pixels when readinginformation from the optical sensor. Moreover, the camera controller 42may also exclude sensor elements within the scope of the lens focus tocreate a rectangular or square array 94 of indexed elements. The indexedarray 94 of sensor elements is within the focused area of the lens andforms the field of view of the working space 74 for the optical sensor40. The region of interest 76 occupies all or a portion of the indexedarray 94 of sensor elements. Although illustrated with the region ofinterest 76 centrally disposed within the field of view of the workingspace 74, it should be appreciated that the region of interest 76 maycomprise alternative portions within the working space 74. Similarly,the region of interest 76 may include larger portions or smallerportions of the working space 74. This improvement allows a largersensor, that is—one having more sensing elements, and thus a lower framerate, to be operated at a higher frame rate by using only parts of thesensor within a limited, defined portion of the array of sensingelements.

FIG. 7 illustrates the above-described relationship in athree-dimensional perspective representation. Camera unit 36 isschematically represented relative to the field of view of the workingspace 74. The total volume 96 of the optical sensor encompasses areduced volume 98 of the region of interest 76. Similar to thedepictions in FIGS. 5 and 6 , the reduced volume 98 is centrallydisposed within the total volume 96. The initial positioning of thecamera unit 36 will impact the relative position of the surgical sitewithin the field of view of the camera. It may therefore be preferablethat a reduced volume 98 be positioned in one or another corner of thetotal volume 96. Alternatively, the reduced volume 98 may take up moreor less of the total volume 96 depending on the size and location of thesurgical site and the distance of the surgical site from the camera unit36. Where the reduced volume 98 occupies a larger proportion of thetotal volume 96, a larger portion of the working space 74 is occupied bythe region of interest 76 on the optical sensor's 40 sensing elements.Similarly, where the reduced volume 98 occupies a particular corner orregion of the total volume 96, a corresponding corner or region of theworking space 74 is occupied by the region of interest 76 on the opticalsensor's 40 sensing elements.

The total volume projection of the working space within the scope of theoptical sensor 40 is related to the hardware configuration of the cameraunit 36. For example, a static lens arrangement may affect the focus fordetermining the closest and farthest observable objects. For example,the observable volume may begin within about 0.5 meters from the cameraunit 36 and may extend to up to 3 meters from the camera unit 36. In analternative example, the observable volume may be at a distance of about0.7 meters to about 2.5 meters from the camera unit 36. The camera unit36 may be configured for an optimal focal distance of the region ofinterest to be from 1 meter to 1.5 meters from the camera unit 36. In analternative example, the camera unit 36 may be configured to have anoptimal focal distance of 1.3 meters from the camera unit.

During navigation in a surgical operation, the navigation computer 26tracks the location and movement of trackers affixed to objects usedduring the surgical operation. The navigation computer 26 may usetwo-dimensional image information received from the optical sensors 40.Optical sensors 40 generate the two-dimensional images from the radiantenergy received at the PCBA of the optical sensor. The intensity ofradiant energy at each active pixel is quantified to generate thetwo-dimensional images processed for navigation tracking. Each activepixel evaluated consumes processing time. It is therefore preferable toreduce the number of active pixels, without otherwise adverselyaffecting image resolution or quality, in order to improve the qualityof accurately tracking rapid movement or very fine movement of a trackedobject. By defining a region of interest as a subset of the total activeelements available within the optical sensors 40, the processing speedmay be increased.

In defining only a portion of the available range of the optical sensor40, it is important to ensure that the region of interest encompassesthe objects to be tracked. At an initial phase of a surgical operation,the navigation system 20 may operate to capture one or multiple imagesof the working space using the full or near-full optical sensor 40range, or using the video camera 41. At the initial phase, high speedand high accuracy tracking may be deemphasized as a surgeon performsinitial setup steps for the surgical operation. Once the surgicaloperation is underway, the surgeon may selectively toggle to switch thenavigation system 20 into a high-speed tracking operational mode.Alternatively, the navigation system 20 may automatically switch betweentracking using the full-range of the optical sensor 40 and a morelimited region of interest. The navigation system 20 may be configuredto switch automatically between operation modes based, for example, onthe detection of the object to be tracked, the determined position,orientation, or movement of the tracked object. The navigation system 20may be configured to switch automatically between operation modes basedon an autonomous movement of the surgical manipulator 56.

During operation, the navigation system 20 may be configured toselectively size and position the region of interest 76 within theoptical sensors' 40 field of view 74. In this way, the navigation system20 can limit the volume of data to be processed and improve trackingspeed. The navigation system 20 may be configured to determine a pose ofeach tracker (e.g., tracker 44, 46, 48) attached to each object ofinterest. Data representative of the identified objects and/or trackers,as well as the pose and movement of each object and/or tracker may bedetermined and stored by the navigation computer 26; or may bedetermined by the camera controller 42 and transmitted to the navigationcomputer 26. The pose and/or movement information of the object and/ortracker may be used to determine the region of interest 76 fornavigation tracking.

For example, the navigation processor 52 may first determine thecoordinates of the individual sensor elements in each optical sensor 40that correspond to the present location of each object of interest.Alternatively, the navigation processor 52 may determine the coordinatesof the individual sensor elements in the video camera 41 sensing arrayin the same way as described above with regard to the optical sensor 40.Because the video camera 41 and the optical sensors 40 are housedtogether within the camera unit 36, the portion of the sensing devicesrespectively within the video camera 41 and the optical sensors 40 witha view of the object correspond to one another. Therefore, determining aregion of interest in the sensing device of the video camera 41 (i.e.the array of sensing elements in which the object appears) informs theregion of interest of optical sensors 40. Accordingly, the active sizeand position of the region of interest 76 within the optical sensors 40can be updated over time for successive tracking cycles by monitoring,for example, where the object is relatively located within the activepixel arrays of the video camera 41.

The navigation processor may reference a table or other data structurestored within memory of the navigation computer 24 or camera controller42. The table may identify which sensor elements are activated orotherwise correspond to various coordinate locations within thelocalizer coordinate system LCLZ. The navigation processor 52 may thenidentify one or more additional sensor elements within a marginsurrounding the sensor elements corresponding to the present position ofthe object. In one embodiment, the navigation process 52 may determinethe number of additional sensor elements within a margin surrounding thepresent position to achieve a total desired proportion of availablesensor elements, for example, 66% of the available sensor elements. Itshould be appreciated that the navigation processor 52 may define eachregion of interest 76 to include any suitable proportion of the totalavailable sensor elements to efficiently and accurately track theobject, taking into account normal or expected movement of the object.The region of interest may be determined independently for each opticalsensor 40.

In some examples, the region of interest 76 may be defined to accountfor movement within a predetermined movement envelope (e.g.,predetermined movement in any one or more of six degrees of freedom fromthe current pose). The expected movement may be based on prior pose data(e.g., a difference in position over time equating with a velocityand/or an acceleration of the object or tracker). In some embodiments,the expected movement may be based on the type of object being tracked.For example, if the tracker is attached to certain portions of theanatomy (e.g., the pelvis, spine, or skull), the tracker may be expectedto move a relatively small amount. If the tracker is attached to otherportions of the anatomy (e.g., the tibia in a robotic knee surgery),then the surgeon may move the anatomy in a large range of motion todetermine joint stability such that the tracker may be expected to movein a circular range of several centimeters to more than a meter.

Accordingly, the navigation system 20 may need to account for thecurrent pose of each tracker and the expected range of motion of eachtracker to set the bounds for the region of interest 76. In oneembodiment, each tracker may have a unique identifier that is detectableby the camera unit 36. For example, each tracker may include a quickresponse (QR) code or other machine-readable code, or each tracker maywirelessly communicate the identifier to the navigation system 12. In analternative example, the user may enter the identifier (e.g., tibiatracker, femur tracker, pelvis tracker, spine tracker, etc.) during aninitial setup phase of the surgical procedure.

Once a current pose of each tracker and the likely range of movement ofeach tracker is determined, the navigation system 20, the cameracontroller 42, and/or the navigation processor 52 can then determine alikely region of interest 76 needed for the camera unit 36. For example,the camera controller 42 or the navigation processor 52 may determine alarge region of interest 76, including a large margin, for example,about 80% of the available sensor elements, if a tibia tracker 46 isused due to the wider range of movement. However, if the trackers areunlikely to move across a large range, then the region of interest 76may be set to a smaller size, for example, about 40% of the availablesensor elements. As a result, the camera controller 42 or the navigationprocessor 52 may dynamically update the region of interest 76 based onthe type of tracker or object being tracked, the expected movement ofthe object or tracker, and/or based on the prior pose data (e.g.,velocity and/or acceleration) of the object or tracker.

The processing of only a subset of the sensor elements from each opticalsensor 40 enables a processing load to be reduced when the navigationprocessor 52 processes the sensing element signals to track the positionand movement of the objects within the operating room. As a result, thelocalizer may sample the light signals received from the trackers 44,46, and 48 at a higher frequency than the localizer 34 might otherwisebe able to sample if the navigation processor 52 were configured toprocess all the available sensor elements within the optical sensors 40.For example, processing the sensing elements within the region ofinterest 76 may occur at a frequency of up to about 1 kHz; whereasprocessing the sensing elements of the entire working space may occur ata frequency of about 300 Hz. The higher frequency processing provided bythe region of interest 76 allows the navigation system to provide higherspeed and higher precision tracking of the objects of interest.

While the embodiments described above are described as being performedby one of the camera controller 42 or the navigation processor 52, itshould be recognized that the identification and determination of theregion of interest 76 and the subset of sensor elements included withinand adjacent to the region of interest 76 may be additionally oralternatively performed by another suitable processor or controller incommunication with the navigation system 20.

Referring to FIG. 8 , an alternative embodiment of a navigation system120 is illustrated with like components numbered the same. In thisembodiment, the navigation system 120 includes camera unit 134 havingthree one-dimensional optical sensors 140, and a full color video camera136. A camera controller 142 is in communication with the opticalsensors 140 and the video camera 136, providing functionality similar tothat described above with regard to camera controller 42. A housing 138supports and houses the optical sensors 140, the video camera 136, andthe camera controller 142.

Referring to FIG. 9 , a block diagram of the localizer's camera unit 134is illustrated with three of the optical sensors 140 being depicted asone-dimensional sensor arrays 202. Such sensors and their arrangementmay be similar to those disclosed in U.S. Pat. No. 6,141,104, the entirecontents of which are hereby incorporated herein by reference. In theillustrated embodiment, the camera unit 134 includes a first sensorarray 204, a second sensor array 206, and a third sensor array 208. Thefirst sensor array 204 may be aligned along a first axis. The secondsensor array 206 may be aligned along a second axis. The third sensorarray 208 may be aligned along a third axis. The camera unit 134 mayinclude any suitable number and arrangement of sensor arrays todetermine a position of a marker, such as a point light source providedby the LEDs 50. Positions may be determined using triangulation methodssuch as those described in U.S. Pat. Nos. 6,141,104 and 6,442,416, theentire contents of which are hereby incorporated herein by reference.

Each sensor array 202 includes a plurality of sensing elements 210. Eachsensing element 210 may correspond to a pixel of a charge coupled device(CCD) or other image sensor. Each sensing element 210 may thus generatean electrical signal (hereinafter referred to as a “sensing elementsignal”) that corresponds to an amount of light incident on that element210. Each sensing element signal is transmitted to the camera controller142 and/or the navigation processor 52 for processing. It should berecognized that additional image processors and/or circuits may bedisposed between the sensing elements 210, the camera controller 42,and/or the navigation processor 52 for processing the sensing elementsignals before being transmitted to the navigation processor 52 in someembodiments.

In one embodiment, an optical filter 220 (more clearly shown in FIG. 10) may be used by the camera unit 134 to determine the position of amarker, such as a point light source provided by an LED 50. The filter220 may include one or more apertures or slits 222 formed therein andmay be positioned in front of each sensor array 202 (i.e., between eachsensor array 202 and the objects within the operating room). Forexample, a single, straight aperture 222 may be used to focus lightemitted from an LED 50 onto a line image (not shown) that is orientedsubstantially perpendicularly to the sensor array 202. The aperture 222may be a long, narrow rectangular aperture within an opaque mask, forexample, and may have an infinite depth of field. Accordingly,disregarding any diffraction effects, the line image may be in sharpfocus regardless of the distance between the LED 50 and the sensorelements 210 of the sensor array 202. As the LED 50 moves along a pathparallel to the longitudinal axis of the aperture 222, the point ofintersection of the line image and the sensor array 202 remainsconstant. An angular field of view of the filter 220 may be changed byvarying the distance between the aperture 222 and the sensor array 202.

Referring to FIG. 10 , the point at which the line image intersects thesensor array 202 is detected by the camera controller 142. For example,the sensor element 210 or elements intersecting the line image areilluminated by the light contained within the line image, and aresulting sensor element signal is generated from each illuminatedsensor element 210. The sensor element signals are received by thecamera controller 142 and/or the navigation processor 52 and are used todetermine the position of the LED 50. For example, in the embodimentillustrated in FIG. 10 , an LED 50 in a first position 230 may cause theline image to illuminate a first sensor element 232 (or group ofadjacent sensor elements). If the LED 50 moves to a second position 234,the resulting line image may illuminate a second sensor element 236 (orgroup of adjacent elements). Similarly, if the LED 50 moves to a thirdposition 238, the resulting line image may illuminate a third sensorelement 240 (or group of adjacent elements). The associated sensorelement signals are transmitted from the illuminated sensor elements 210to the camera controller 142 and/or navigation processor 52 to determinethe associated position of the LED 50 as described above. In the eventthat positions 230, 234, and 238 correspond to the expected movement ofan object, a resulting window (described below) for tracking themovement of the object would be defined to include sensor elements 232,236, and 240 as well as any suitable number of adjacent sensor elements210 corresponding to the size of the object and the expected movement ofthe object.

The navigation processor 52 applies a dynamic window 212 to each sensorarray 202 to selectively enable and disable the processing of thesensing element signals provided by each sensor array 202. Each window212 represents a subset of sensing elements 210 that will be processedor used by the navigation processor 52 to identify and track thelocation of the objects within the operating room. Accordingly, theapplication of the dynamic window 212 to each sensor array 202effectively crops the usable sensing elements 210 of each sensor array202. Thus, only the sensing elements 210 that are identified as beingwithin the window 212 are processed by the navigation processor 52 toidentify and track the pose of one or more objects within the operatingroom.

Each window 212 may be identified by the navigation processor 52 or thecamera controller 142 based on signals that identify a location of oneor more objects within the operating room, for example. In oneembodiment, the camera controller 142 or the navigation processor 52 maybe used to quickly identify objects of interest in the operating room aswell as their general location within the room using the video camera136. Thus, the machine vision system 12 may provide relativelylow-resolution tracking of the objects within the operating room basedon the machine vision information. The navigation system 20, on theother hand, may provide relatively high-resolution tracking of theobjects within the operating room using the optical sensors 140.

During operation, the navigation computer 26 identifies one or moreobjects of interest within the operating room and determines a pose(i.e., position and/or orientation) of each object. Additionally, oralternatively, the navigation computer 26 may determine a pose of eachtracker (e.g., tracker 44, 46, or 48) attached to each object ofinterest since each object of interest will typically include a tracker.Data representative of the identified objects and/or trackers, as wellas the pose of each object and/or tracker, is transmitted from thecamera controller 142 to the navigation processor 52. Since the trackersare the components that are directly tracked by the camera controller142, rather than the objects themselves, the pose of the trackers may beused to determine which sensor elements 210 to enable or disable asdescribed herein.

The navigation processor 52 receives, from the camera controller 142,data representative of an identification of the objects of interest thatare determined by the navigation computer 26 to be present within theoperating room and data representative of the pose (i.e., positionand/or orientation) of each object and/or tracker within the localizercoordinate system LCLZ. The navigation processor 52 then makes adetermination of what portions of each sensor array 202 to process inorder to efficiently track the pose of each object and/or tracker.

For example, the navigation processor 52 may first determine whichsensing elements in each sensor array 202 correspond to the presentlocation of each object. To do so, the navigation processor 52 mayreference a table or other data structure stored within memory of thenavigation computer 26 or camera controller 142. The table may identifywhich sensing elements 210 are activated or otherwise correspond tovarious coordinate locations within the localizer coordinate systemLCLZ. The navigation processor 52 may then identify one or moreadditional sensing elements 210 within each sensor array 202 that areadjacent to (i.e., on either or both sides of) the sensing elements 210corresponding to the present position of the object. In one embodiment,the navigation processor 52 may determine the number of additionalsensing elements 210 adjacent to the sensing elements 210 correspondingto the position of the object to be equal to 100% of the sensingelements 210 corresponding to the position of the object. The navigationprocessor 52 may then determine the window 212 for each sensor array 202to include the sensing elements 210 corresponding to the presentposition of each object as well as the additional sensing elements 210determined above. Thus, in this example, the navigation processor 52 maydefine the window 212 for each sensor array 202 to be equal to 3 timesthe number of sensing elements 210 corresponding to the size andposition of the object.

It should be recognized that the navigation processor 52 may define eachwindow 212 to include any suitable number of sensing elements 210 toenable each object to be efficiently and accurately tracked, taking intoaccount normal or expected movement of the object. It should also berecognized that the navigation processor 52 may identify a differentnumber of sensing elements 210 to be included within the window 212 foreach sensor array 202. Accordingly, the window 212 may be defined toaccount for movement of the object beyond its current pose. In somecases, the window 212 may be defined to account for movement within apredetermined movement envelope (e.g., predetermined movement in any oneor more of six degrees of freedom from the current pose). The expectedmovement may be based on prior pose data (e.g., a difference in positionover time equating with a velocity and/or an acceleration of the objector tracker) in some embodiments, or may be based on the type of objectbeing tracked.

For example, if the tracker is attached to certain portions of theanatomy (e.g., the pelvis or spine), the tracker may be expected to movea relatively small amount. If the tracker is attached to other portionsof the anatomy (e.g., the tibia in a robotic knee surgery), then thesurgeon may move the anatomy in a large range of motion to determinejoint stability such that the tracker may be expected to move in acircular range of several inches to several feet. Accordingly, thenavigation computer 26 may need to account for the current pose of eachtracker and the expected range of motion of each tracker. In oneembodiment, each tracker may have a unique identifier that is alsodetectable by the navigation computer 26. For example, each tracker mayinclude a quick response (QR) code or other machine-readable code, oreach tracker may wirelessly communicate the identifier to the navigationcomputer 26. Alternatively, the user may enter the identifier (e.g.,tibia tracker, femur tracker, pelvis tracker, spine tracker, etc.)during an initial setup phase of the surgical procedure.

Once the current pose of each object's tracker and the likely range ofmovement of each object's tracker is determined, the navigation computer26 and/or the camera controller 142 can then determine a likely field ofview needed for the camera unit 134 of the navigation system 120. Thewindows 212 may then be based on this field of view. For example, thecamera controller 142 or the navigation processor 52 may increase allwindows 200% if a tibia tracker 46 is used since all windows 212 need tobe able to encompass the tibia tracker 46. However, if all trackers areunlikely to move a large amount, then the windows 212 can be set to asmaller size. As a result, the camera controller 142 or the navigationprocessor 52 may dynamically update the windows 212 based on the type oftracker or object being tracked, the expected movement of the object ortracker, and/or based on the prior pose data (e.g., velocity and/oracceleration) of the object or tracker.

It should be recognized that each window 212 may be different for eachsensor array 202. Thus, in one embodiment, the window 212 for the firstsensor array 204 may include a first number of sensing elements 210corresponding to the position of the objects, the second sensor array206 may include a different, second number of sensing elements 210corresponding to the position of the objects, and the third sensor array208 may include a different, third number of sensing elements 210corresponding to the position of the objects.

As described herein, the processing of only a subset of sensing elements210 from each sensor array 202 enables a processing load to be reducedwhen the navigation processor 52 processes the sensing element signalsto track the position of the objects within the operating room. As aresult, the localizer 34 may sample the light signals received from thetrackers 44, 46, 48 at a higher frequency than the localizer 34 mightotherwise be able to sample if the navigation processor 52 wasconfigured to process all sensing element signals from all sensingelements 210.

While the embodiments herein are described as being performed by thenavigation processor 52, it should be recognized that the identificationand determination of the windows 212 and the subsets of sensing elements210 included within and adjacent to the windows 212 may be additionallyor alternatively performed by the camera controller 142 or anothersuitable processor or controller.

Referring to FIG. 11 , in one embodiment, each dynamic window 212 may beimplemented using an array 302 of bit masks 304. Each bit mask 304 ofthe bit mask array 302 corresponds to sensing element data 306 output byan individual sensing element 210 of an individual sensor array 202.Thus, each bit mask 304 of the bit mask array 302 may cause thenavigation processor 52 to make a decision to enable or disable theprocessing of data from a respective sensing element 210. For example,if the navigation processor 52 (or another processor) stores a bit valueof 1 in the bit mask 304, the navigation processor 52 may enable theprocessing of the data from the associated sensing element 210.Similarly, storing a bit value of 0 may disable the processing of thedata from the associated sensing element 210. The bit mask array 302 foreach sensor array 202 may be stored in the memory of the navigationcomputer 26 as one or more data structures.

Referring to FIG. 12 , in an alternative embodiment, the bit masks maybe implemented in hardware as an array 402 of transistors or othergating devices 404 that selectively enable or disable each sensingelement signal from reaching the navigation processor 52. Thus, eachgating device 404 may be controlled by the navigation processor 52 (oranother processor such as the camera controller 142) to enable orprevent the sensing element data from being processed by the navigationprocessor 52. For example, if the navigation processor 52 (or anotherprocessor) activates the gating device (i.e., by enabling the gatingdevice to conduct), the gating device 404 may enable the data from theassociated sensing element 210 to be transmitted to the navigationprocessor 52. Similarly, deactivating the gating device 404 may preventthe data from the associated sensing element 210 from being transmittedto the navigation processor 52. The gating devices 404 may be positionedwithin the camera unit 36 or another suitable portion of the navigationsystem 20 in an electrical path between the sensing elements 210 and thenavigation processor 52.

FIG. 13 is a flow diagram of a method 500 for tracking objects within anoperating room. For example, the method 500 may be used to tracksurgical instruments 22 and other tools that a surgeon may use tooperate on a patient, as well as tracking the patient's anatomy. Themethod includes receiving image data of the operating room. The methodincludes identifying one or more objects of interest in the operatingroom. The method includes identifying the position of each object ofinterest in the image data. The method includes determining the sensingelements corresponding to the identified position of each object. Themethod includes defining a subset of sensing elements in each sensorused for tracking each object of interest. The method includes usingonly the subset of sensing elements to track the object within theoperating room.

In an embodiment, each step of the method 500 may be implemented as oneor more computer-executable instructions that are stored within one ormore computer-readable media. In a specific embodiment, the method 500may be implemented using the navigation system 20 shown in FIG. 1 , orthe navigation system 120 shown in FIG. 8 . For example, the cameracontroller 42, 142, and/or the navigation processor 52 may executeinstructions stored within memory of the camera controller 42, 142and/or the navigation system 20 to perform the steps of the method 500described herein.

In one embodiment, the method 500 includes receiving 502 image data ofthe operating room from one or more optical sensors 40 or machine visioncamera 36. For example, in a partial or total knee replacement surgery,the optical sensors 40 or machine vision camera 36 may generate imagedata of the surgeon, the patient, the trackers 46, 48 attached to thepatient's femur F and tibia T, respectively, the surgical instrument 22,and the tool tracker 48, among others. The method includes identifying504 one or more objects of interest from the image data. The objects ofinterest may be defined in a similar manner as described above withreference to disclosed embodiments. In the example of a knee replacementsurgery, the objects of interest may include the surgical instrument 22,the femur F, the tibia T, and the trackers 44, 46, 48. In oneembodiment, identifying each tracker 44, 46, 48 includes using anidentifier unique to that tracker as described above. The method mayalso identify 506 a position of each object in the image data asdescribed above.

The method may also determine an expected movement or change in pose ofeach object in a similar manner as described above. For example, themethod may use a lookup table or another suitable data structure storedin memory to correlate the type of object with an expected range ofmotion or change in pose. Additionally, or alternatively, the method mayreference prior pose data of each object to determine a velocity,acceleration, and/or expected change in pose of each object. The methodmay then determine the expected movement or change in pose of eachobject.

The navigation processor 52 may also determine 512 which sensor elementsof optical sensor 40 or sensing elements 210 of each sensor array 202,within the camera unit 36, 134 correspond to the location of each objectwithin the localizer coordinate system LCLZ.

The navigation processor 52 may then determine 514 a subset of sensorelements of optical sensor 40 or sensing elements 210 within each sensorarray 202 that will be used to track each object. For example, thenavigation processor 52 may determine each region of interest 76 orsubset of sensing elements 210 to include the additional elementsdetermined in step 512 as well as a predetermined number of elementsdetermined in step 512. These elements may be defined as being includedin a region of interest 76 or window 212 that may be dynamically updatedbased on new data received and/or new data determined by the navigationprocessor 52. As noted above, the region of interest 76 or windows 212may be dynamically updated to include the elements corresponding to theexpected movement or change of pose of each object. When the navigationprocessor 52 has determined each subset of elements in step 514, thenavigation processor 52 uses 516 only the subset of elements to trackeach identified object within the operating room. In one embodiment, thenavigation processor 26 and/or the camera controller 42, 142 only readsout information from sensor elements within the region of interest 76 orwindow 212. In one embodiment, the navigation processor 52 uses a bitmask array 302 such as described in FIG. 6 or an array 402 of gatingdevices such as described in FIG. 7 to process only the subset ofsensing elements 210. Alternatively, the navigation processor 52 may useany suitable device or technique to process only the selected subset ofelements for tracking the objects.

Accordingly, as described herein, the method may be used to identifyeach object of interest within a space, such as an operating room. Thenavigation system 20, including the localizer 34 and camera unit 36, 134provide high speed, high fidelity tracking of the objects. Thenavigation system 20 may accomplish this by only activating the elementswithin one or more dynamically defined regions of interest 76 or windows212 corresponding to the position and/or expected movement of eachobject while deactivating the elements that are not included within theregion of interest 76 or windows 212. As a result, the navigationprocessor 52 and/or the camera controller 42, 142 may benefit from areduced processing workload resulting from the reduced number of sensorelement signals needing to be processed to track the objects.

Several embodiments have been discussed in the foregoing description.However, the embodiments discussed herein are not intended to beexhaustive or limit the invention to any particular form. Theterminology that has been used is intended to be in the nature of wordsof description rather than of limitation. Many modifications andvariations are possible in light of the above teachings and theinvention may be practiced otherwise than as specifically described.

What is claimed is:
 1. A surgical navigation system for tracking an object within an operating room, the surgical navigation system comprising: a camera unit comprising: a housing; a first optical sensor coupled to housing and comprising sensing elements adapted to sense light in a near-infrared spectrum; a second optical sensor coupled to the housing and being adapted to sense light in a visible light spectrum; and a controller in communication with the first and second optical sensors, and wherein the controller is configured to: obtain, from the second optical sensor, data related to the object within the operating room; and modify control of the sensing elements of the first optical sensor based on the data of the object obtained by the second optical sensor.
 2. The surgical navigation system of claim 1, wherein the data related to the object within the operating room comprises one or more of (1) a type of the object (2) a pose data of the object, and/or (3) an expected movement of the object.
 3. The surgical navigation system of claim 2, wherein the controller is configured to modify control of the sensing elements to activate a subset of the sensing elements, the subset being less than all of the sensing elements.
 4. The surgical navigation system of claim 2, wherein the controller is configured to modify control of the sensing elements to prevent processing of a subset of the sensing elements, the subset being less than all of the sensing elements.
 5. The surgical navigation system of claim 2, wherein the controller is configured to modify control of the sensing elements to adjust a size of a field-of-view viewed by the first optical sensor.
 6. The surgical navigation system of claim 2, wherein the controller is configured to modify control of the sensing elements to define a size and/or position of a region-of-interest viewed by the first optical sensor, the region-of-interest being a sub-region within a full field-of-view of the first optical sensor.
 7. The surgical navigation system of claim 2, wherein the controller is configured to modify control of the sensing elements by adjusting a frequency by which the sensing elements are read-out.
 8. The surgical navigation system of claim 2, wherein the pose data of the object includes one or more of: a velocity of the object and/or an acceleration of the object.
 9. The surgical navigation system of claim 1, wherein the camera unit comprises two first optical sensors coupled to the housing and each comprising the sensing elements adapted to sense light in the near-infrared spectrum.
 10. The surgical navigation system of claim 9, wherein the camera unit comprises: a rigid support structure disposed within the housing and being configured to support the two first optical sensors, and wherein the two first optical sensors are spaced apart from one another on the rigid support structure; and the second optical sensor rigidly supported by the housing relative to the two first optical sensors in a predefined relationship.
 11. The surgical navigation system of claim 9, wherein the housing has an elongated shape defining a first end and an opposing second end, and wherein one of the first optical sensors is disposed adjacent to the first end of the housing and the other one of the first optical sensors is disposed adjacent to the opposing second end of the housing.
 12. A method of operating a surgical navigation system for tracking an object within an operating room, the surgical navigation system comprising a camera unit comprising a housing, a first optical sensor coupled to housing and comprising sensing elements adapted to sense light in a near-infrared spectrum, a second optical sensor coupled to the housing and being adapted to sense light in a visible light spectrum, and a controller in communication with the first and second optical sensors, the method comprising the controller performing the steps of: obtaining, from the second optical sensor, data related to the object within the operating room; and modifying control of the sensing elements of the first optical sensor based on the data of the object obtained by the second optical sensor.
 13. The method of claim 12, comprising the controller obtaining the data related to the object as being one or more of: (1) a type of the object (2) a pose data of the object, and/or (3) an expected movement of the object.
 14. The method of claim 13, comprising the controller modifying control of the sensing elements by activating a subset of the sensing elements, the subset being less than all of the sensing elements.
 15. The method of claim 13, comprising the controller modifying control of the sensing elements by preventing processing of a subset of the sensing elements, the subset being less than all of the sensing elements.
 16. The method of claim 13, comprising the controller modifying control of the sensing elements for adjusting a size of a field-of-view viewed by the first optical sensor.
 17. The method of claim 13, comprising the controller modifying control of the sensing elements for defining a size and/or position of a region-of-interest viewed by the first optical sensor, the region-of-interest being a sub-region within a full field-of-view of the first optical sensor.
 18. The method of claim 13, comprising the controller modifying control of the sensing elements by adjusting a frequency by which the sensing elements are read-out.
 19. The method of claim 13, comprising the controller obtaining the pose data of the object as being one or more of: a velocity of the object and/or an acceleration of the object.
 20. A camera unit for tracking an object within an operating room, the camera unit comprising: a first optical sensor comprising sensing elements adapted to sense light in a near-infrared spectrum; a second optical sensor adapted to sense light in a visible light spectrum; and a controller in communication with the first and second optical sensors, and wherein the controller is configured to: obtain, from the second optical sensor, data related to the object within the operating room; and modify control of the sensing elements of the first optical sensor based on the data of the object obtained by the second optical sensor. 